A Study of the Effects of Usability on Risk Perception and User Affinity of Risk Finder

In October 2013, Climate Central, a non-profit research organization, released Risk Finder, an online, interactive mapping tool that allows users to explore the effects of rising sea level in a specific geographic region. A research study for client Dan Rizza of Climate Central was conducted on the usability of the Risk Finder tool in an effort to identify potential system errors, improve user experience, and assess the future use of this tool based on user perception. Primary investigators Dr. Daniel Richards and Mrs. Megan McKittrick utilized an approach known as “productive usability,” which allows researchers to openly observe participants engaged in a talk-aloud protocol whereby users articulate what they are thinking and feeling as they are performing a particular task or using a particular product. Qualitative data was obtained from a group of four voluntary participants through pre-interviews, observation, and post-interviews while these users explored Risk Finder. Undergraduate students from Old Dominion University’s ENGL 231C class, an introductory Scientific and Technical writing course, were included in this study as observers and transcribers, as this service-learning project allowed students the opportunity to participate in research. During this research, it was valuable to observe the relationship between the usability of the Risk Finder tool and, based on observation of emotional and verbal reactions, how this usability affected the participants’ risk perception and affinity for the Risk Finder application, particularly during the postinterview sessions. Based on analysis of qualitative data gathered from the usability study, Risk Finder displayed a shortfall in usability. Additionally, the data analyzed strongly suggest that usability may play a role in the perception of risk associated with climate change via visual communication. A comprehensive assessment of parameters and previous research indicates that users’ emotional responses to sea-level rise may influence their perceptions of the product’s usability. 1 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 ith the exponential rate of sea level rise, it is of the utmost importance that the public is informed of the imminent dangers associated with climate change. Climate Central, a non-profit research organization, has developed an online, interactive tool titled Risk Finder to fulfill this need. Dr. Daniel Richards and Mrs. Megan McKittrick of Old Dominion University, along with the students of Honors English 231C, conducted a usability test of the Risk Finder tool with participants recruited from the local Hampton Roads community. As part of the overall analysis, this report contains a review of current literature in which different aspects of product design and their relation to user emotion are defined. The importance of user-centered design is also discussed, and the impact of personal relevance is explored. This report goes on to examine different facets of usability and their influence on risk perception and user affinity for the Risk Finder tool, as well as the role that emotion may play in the users’ overall determination of usability. W 2 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 LITERATURE REVIEW Research on Emotional Design The inherent design and usability of a product have been proven to affect the emotions of the user. Subsequently, the user draws upon emotion as a critical factor when assessing a product’s significance. Norman (2004) defines three major components of effective product design: usability, usefulness, and aesthetic. Usability is contingent on the ability of the user to engage with the product in the way the designer intended (Norman, 2004). Usefulness depends on how relevant the user finds this product to be in congruence with his or her goals, and aesthetic refers to how attractive a product is (Norman, 2004). Norman (2004) explains that the user's opinion of a product, positive or negative, depends on the success or failure of these three design aspects. The connection between product usability and user emotion is further solidified in Jokinen’s (2014) later study of user experience and emotional states. In this study, Jokinen (2014) attempts to apply 3 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 emotional theory to user experience as defined by an individual’s response to using a product. To do this, Jokinen (2014) conducted a usability test of several basic computer programs during which users would perform designated tasks in an allotted time. Jokinen (2014) found that there is a strong link between a user’s emotions before and after using a product, and these emotions affect task performance. Norman (2004) suggests that product design and usability affects emotion during and after the use of a product, whereas Jokinen (2014) concludes that emotions before a product is used greatly affect opinions on the product's usability. The Benefits of User-Centered Design on Site Usability When designing a website or online tool for a target audience, effective usability is achieved as the audience’s needs are uncovered and implemented through user-centered design. In Becker and Yannotta’s (2013) study of the redesign process of a university 4 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 library website, iterative usability testing, a process of incorporating user feedback into a continuous redesign process, allowed the needs of the target audience to be met, resulting in a more attractive, usable, and useful website (Becker and Yannotta, 2013). Retchless’s (2014) study of the perception of uncertainty in cartographic climate communication continues this discussion. Through the analysis of two climate change mapping tools, NOAA’s Sea Level Rise and Coastal Flooding Impacts Viewer and Climate Central’s Surging Seas, Retchless (2014) explains that climatechange communication is most effective when the designer tailors to the needs of the user, especially by adding customizable variables appropriate to the user, a concept that will be discussed further below. Both Becker and Yannotta’s (2013) and Retchless’ (2014) studies come to a similar conclusion: user-centered design is essential when creating an informative website. 5 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 The Use of Visuals and Personal Relevance in Risk Communication When communicating risk, visuals provide a greater impact on the audience than numerical data (Braasch, 2013; Retchless, 2014). This effect on the audience is an important aspect of emotional design, as the purpose of these visuals is to incite an emotional response from the viewer. These claims are supported in Braasch’s (2013) study of the efficacy of using visuals to communicate climate change. In his article, Braasch (2013) highlights the importance of visuals in risk communication, particularly their effectiveness over written or spoken information. Retchless’ (2014) study on uncertainty in sea level rise projection maps continues this narrative. Retchless (2014) states that the use of local roads when depicting water inundation is highly impactful on the user of these tools, such as in Climate Central’s Risk Finder. In these studies, Braasch (2013) focused on images of climate change, while Retchless (2014) studied interactive maps. Regardless, both studies found that, when communicating climate change risk, 6 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 visuals are impactful, especially when personally meaningful to the audience. Linking Product Design and Usability to User Response When these concepts of usability, user-centered design, and personal relevance are linked, the user forms perceptions of gathered information. These perceptions ultimately determine how the user will respond to a given issue. In Ancker, Chan, and Kukafka’s (2009) study, these concepts are researched together in a usability study of an interactive tool communicating health risks. Users were able to input their own health risk factors, a trait that is relatable to Climate Central’s Risk Finder, in which users enter addresses and modify sea level height (Ancker et. al., 2009). The usability test, conducted on a small group of volunteers from the community, allowed users to explore and manipulate the website while vocalizing their observations (Ancker et. al., 2009). This methodology parallels the Risk Finder usability test, which allows 7 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 volunteers to openly interact with Risk Finder and vocalize observations. The findings of this study are telling: users enjoyed the interactive elements of the web tool and found aspects such as personalized risk factors and visual information to be impactful, relevant, and understandable—more so than numbers and graphs—a conclusion that is supported by the research conducted by Braasch (2013) and Retchless (2014) on the impact of visuals when communicating climate change risk (Ancker et. al., 2009). The researchers also noted that when using the tool, participants exhibited distinct emotional responses when provided with risk information, such as the likelihood of experiencing a heart attack (Ancker et. al., 2009). With these conclusions, it can be inferred that the reactions to Risk Finder, a similar tool in purpose and technology, may be comparable. With that being said, Ancker, Chan, and Kukafka’s (2009) study is that of communicating health risk, not climate change risk, which may call for different forms of 8 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 communication as they are different disciplines. As investigated in this review, there is a strong connection between the emotional design of a product, user-centered design, and personal relevance, as supported by numerous existing studies. These studies conclude that the implementation of these elements has a great effect on how an audience perceives and responds to information being presented. With this in mind, during the usability test of Climate Central’s Risk Finder tool, it is important that user emotion and response are thoroughly observed and recorded in order to confirm their relationship in risk communication concerning climate change. METHODS In this IRB-approved productive usability study of Climate Central’s Risk Finder tool, volunteers were solicited to represent the greater Hampton Roads population via systematic sampling. Invitations were mailed to 250 Hampton Roads addresses obtained from Polk directories in the Old Dominion University library with flyers 9 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 requesting contact information and available times for testing. Four volunteers were selected and served as the sample being tested. As this is a service-learning project, students from Old Dominion University served as observers throughout the study, during which participants engaged in talk-aloud protocol while openly exploring the Risk Finder tool. Pre-test interviews were conducted, which focused on gathering qualitative data associated with risk perception, the participant’s experiences living in Hampton Roads in reference to climate change and flooding, and the participant’s expectations of a risk communication website like Risk Finder. After testing, post-interviews were conducted regarding the participant’s risk perception and opinions after using the tool. Each instance of testing was recorded using both audio and active screen capture (Camtasia). After each test concluded, the student observers transcribed the audio recordings and coded for themes associated with technical issues, risk communication events, risk perception, and the potential impact of associated risk. 10 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 RESULTS Based on the analysis of qualitative data gathered from the usability study, users were unable to utilize all features in the Risk Finder tool, which led to a decreased affinity for the application. A comprehensive assessment of parameters and previous research indicates that user emotion may influence usability. Procedure Before analyzing the data, two dependent variables were established: the user’s perception of climate change and the risks associated with it, as well as the user’s affinity for the Risk Finder tool. These dependent variables were anticipated to be affected by an all-encompassing independent variable: the usability of Risk Finder. Usability was further broken down into six individual emotion-based factors for coding purposes—navigation, able to meet objective, unable to meet objective, negative response, positive response, visuals meaningful, and visuals not meaningful— based on user feedback derived from the audio recording 11 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 transcriptions. The navigation independent variable was later omitted as there were not enough instances to be considered statistically significant. Each instance of an emotion-based factor was tallied and recorded. User Response The first facets of user response recorded were the “able to meet objective” and “unable to meet objective” categories. For example, one user voiced a desire to “click on Shore Drive.” If the user was able to navigate to “Shore Drive,” the instance would be labeled as “able to meet objective.” If unable to carry out the action successfully, the instance was labeled as “unable to meet objective.” As seen in Figure 1, 84% of objectives set by the user were successful, while 16% were not. This suggests a relatively high degree of usability in this respect. Figure 1: Ability to Meet Personal Objective VVV 12 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 Vocalized user comments tell a different story. When a user vocalized a comment that was positive in nature, such as “that’s interesting,” the instance was labeled as “positive comment.” On the other hand, when a user made a negative comment, such as “this is kind of hard to see,” the instance was labeled as “negative comment.” As seen in Figure 2, 84% of the comments were negative in nature, while only 16% were positive. The final aspect of user response examined was the users’ reactions to the visuals. Each time a user encountered a graphic, such as data tables or text, the user indicated his or her level of understanding of the present information. In 78% of the instances recorded, the user found displayed information to be confusing or “not meaningful,” compared to the 22% 13 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 of instances that were found “meaningful” (Figure 3). This data suggests that to a certain degree, users are not finding the data tables effective. Risk Finder’s Effect on User Risk Perception of Climate Change Users were asked to label their level of concern regarding climate change before and after using Risk Finder from a list of four terms: alarm, concern, skeptical, or other. As shown in Figure 4 below, the users’ risk perceptions of climate change were unchanged after using Risk Finder. 14 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 User Affinity for Risk Finder One important measure of usability is whether or not the participant intends to use the product again. With the conclusion of each usability test, users were asked whether they intended to use Risk Finder in the future. As seen in Figure 5, 50% of the users replied “would not use,” 50% replied “might use,” and 0% replied “would use.” Meeting Individual Objectives versus Perception of Overall Ability As discussed in the previous User Response section, 84% of vocalized objectives set by all users were able to be met (Figure 1). During the post-test interviews, users were asked if they felt as if they “were able to do what [they] needed to do,” referring to the 15 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 objectives announced by participants during productive usability testing. Contrary to the Personal Objectives data, 3 out of 4 users said they were, overall, unable. (Figure 6). This is a significant discrepancy in the data, which will be further examined and discussed in the following section. USER A Objective Set Able /Unable Overall: Able/Unable Navigate to "Map" Unable Able Zoom Able Scrolling to Find House Able Click Map to Zoom Unable Find Slider Information Able Navigate to "Forecast" Able Enter in Information Able Return to Previous Page Able Search "Virginia Beach" Able Adjust Slider Able Zoom Able Zoom Able Scrolling to Find Son's House Able Zoom Able Adjust Slider Able Navigate to "Comparison" Able Click "Buildings" Able Navigate to "Fast Look" Able Navigate to "More" Able Navigate to "Science" Able Navigate to "Widgets" Able 16 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 USER B Objective Set Able/Unable Overall: Able/Unable Navigate to "Forecast" Able Unable Navigate to "Map" Able Navigate to "Virginia Beach" Able Zoom Unable Zoom Able Click on Map to Zoom Unable Navigate to "Comparison" Able Navigate to Previous Page Able Navigate to "State" Able Navigate to "Map" Able Search "Virginia Beach" Unable Click on Map to Zoom Unable Zoom Able Zoom to Street Level Unable Zoom Out Able Navigate to "Comparison" Able Navigate to "Help" Able Navigate to "Science" Able Navigate to Previous Page Unable Navigate to "Major Expansion" Able Navigate to "Surface Flooding" Able USER C Objective Set Able/Unable Overall: Able/Unable Zoom on Main Page Image Unable Unable Navigate to Main Map Able Scroll on Map Able Click "Social Vulnerability" Able Find Neighborhood Able Adjust Slider Able Click "Property" Able Zoom Out Able Find ODU Able Click "Features" Able Navigate to "Roads" Able Click "Social Vulnerability" Able Adjust Slider Able Navigate to "Help" Able Return to Previous Page Able Search Zipcode Unable Search Zipcode Unable 17 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 USER D Objective Set Able/Unable Overall: Able/Unable Navigate to "Norfolk" Able Unable Click "Social Vulnerability" Able Click "Population" Able Click "Ethnicity" Able Click "Show Features" Able Navigate to "Forecast" Able Navigate to "Virginia Beach" Able Navigate to "Hampton" Able Navigate to "Norfolk" Able Navigate to "Fast Look" Able Figure 6: Individual User Objectives and Perception of Overall Ability DISCUSSION In terms of usability, the majority of the goals set by users were met during open exploration of the application (Figure 1). This majority indicates a degree of usability in Risk Finder tool; if users are able to accomplish tasks, it can be inferred that the system is usable to some degree. However, as mentioned in the preceding section, 3 out of 4 users claimed in the post-test interviews that they were unable to accomplish what they wanted to accomplish. These findings suggest that, while the application was usable, the functionality of the tool did not meet the user’s expectations. This discrepancy raises a question: what influenced the users’ 18 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 perceptions of usability? Based on the vocalized feedback, which was predominately negative (Figure 2), it appears that it was emotion that altered the user's ability to complete personal tasks and the overall usability of Risk Finder. In this study, the data indicates that the negative emotional responses correlate to a negative perception of usability. This concept is supported by both Norman's (2004) findings on the impact of product design on user emotion and Jokinen’s (2014) study on the relationship between user emotion and task performance. This hypothesis leaves an opening for further research, as a larger sample size would permit a greater breadth of emotional feedback, which may support this finding. In regards to risk perception, there is not enough data to sufficiently make the claim that t h e usability of a risk communication tool impacts the perception of risk. However, it is apparent that the users’ perceptions of risk associated with climate change did not alter after using Risk Finder (Figure 4) and therefore, 19 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 there is a distinct possibility that usability has played a significant part. Risk Finder uses visual representations to communicate risk associated with climate change, a method that has been effectively proven to bolster effective risk communication (Retchless, 2014). Based on this study’s findings, the participants were not gaining meaningful information from the visuals by a vast majority (Figure 3). This does not show that usability affects risk perception, but it is certainly within the realm of possibility. Due to the size of the sample, there is ample qualitative data that suggests usability has a significant effect on the user’s opinion of a product. Eighty-four percent of all vocalized feedback was negative in nature. Further, according to a list of recorded concerns, Risk Finder was f o u n d t o b e “too complicated” and “too slow.” Additionally, not one of the users claimed they would use Risk Finder personally; half of the participants said “would not use” and the other half said “might use.” This suggests that users attribute the Risk Finder’s usability in its current state to the unlikelihood that 20 OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 they would use it in future. The findings of this study may prove to be influential not only in risk communication of sea level rise, but also in crisis communication outside of the sphere of climate change. Understanding the relationship between emotion and the perception of usability may allow for the creation of more effective applications, which may well aid the efficacy of risk depiction. With the imminent dangers of climate change looming in the future, tools such as Risk Finder are becoming increasingly necessary, and it is vital that these tools are usable for all members of the public. 21 Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015 ReferencesAncker, J. S., Chan, C., & Kukafka, R. (2009). Interactive Graphics forExpressing Health Risks: Development and QualitativeEvaluation. Journal of Health Communication, 14(5), 461-475. doi:10.1080/10810730903032960Becker, D. A., & Yannotta, L. (2013). Modeling a LibraryWebsite Redesign Process: Developing a User-Centered Website Through Usability Testing.Information Technology & Libraries, 32(1), 6-22.Braasch, G. (2013). Climate change: Is seeing believing?. Bulletin ofThe Atomic Scientists, 69(6), 33-41.doi:10.1177/0096340213508628Jokinen, J. P. (2014). Emotional user experience: Traits, events, andstates. International Journal of Human-Computer Studies,7667-77. doi:10.1016/j.ijhcs.2014.12.006Norman, Donald A. (2004). Emotional design: Why we love (orhate) everyday things. New York: Basic Books. Retrievedfromhttp://proquest.safaribooksonline.com.proxy.lib.odu.edu/book/design/9780465004171Retchless, D. P. (2014). Sea Level Rise Maps: How IndividualDifferences Complicate the Cartographic Communication ofan Uncertain Climate Change Hazard. CartographicPerspectives, (77), 17-32. doi:10.14714/CP77.1235 22OUR Journal: ODU Undergraduate Research Journal, Vol. 3 [2015], Art. 4 http://digitalcommons.odu.edu/ourj/vol3/iss1/4 AcknowledgmentsI would like to thank Dr. Daniel Richards and Megan McKittrick fortheir mentoring and encouragement throughout this process, as wellas my teammates, the students of the Spring 2015 ClimateCommunication URLC, for their collaboration. I would especially liketo thank Climate Central for their support of service learning,dedication to research, and partnership throughout this project. 23Jacobson: A Study of the Effects of Usability on Risk Perception Published by ODU Digital Commons, 2015